Field of the Invention
The invention is related to image processing of captured images. More particularly, it is related to multi-band denoising of images.
Description of the Related Art
The image signal processor (ISP) takes the raw image from the image sensor, and then optionally performs one or more of several operations, such as: gain, binning (in low-light), noise reduction, local tone mapping, demosaicing, white balancing, gamma, filtering, and color enhancement. The ISP provides a YCbCr (i.e., luma-chroma) image, which is later compressed. In most cases, post-ISP operations are performed in YCbCr space, not in RGB (i.e., red-green-blue) space. There are several challenges associated with obtaining an accurate YCbCr noise model. First, the noise characteristics of images obtained by a digital camera are quite complicated. They may have signal dependence, e.g., due to shot noise in the image sensor and gamma operations in the camera pipeline, frequency dependence caused by demosaicing, luma sharpening, chroma band-limiting, and binning, inter-channel correlation resulting from demosaicing and color correction, and channel dependence caused by white balancing. In short, the noise may have signal-wise, channel-wise, and band-wise dependencies.
The noise reduction pipeline in a typical consumer digital camera, e.g., a mobile phone camera, is fairly basic. It is constrained by the number of delay lines available for the image signal processor, as well as computational limitations. Second, since it typically takes a few years to design, test, and produce an image signal processor; the noise reduction algorithm is typically a few generations old by the time a device makes it to market. The camera pipeline may introduce a number of artifacts, such as false edges, sprinkles, and black/white pixel clumps that, from a signal point-of-view, are not noise, but actually appear more like structure. These artifacts severely degrade image quality in bright light, especially in the sky regions (i.e., “blue-sky noise”), but they are especially severe in low-light. One way to mitigate noise as well as artifacts is to increase exposure time so that more photons can be accumulated in the sensor, but this introduces motion blur. Another way to mitigate noise is to use a sensor that has larger well capacity, such as four-thirds, APS-C, or full frame. These sensors are used in DSLR cameras and are quite expensive. Also, they are physically larger, requiring more space and making them infeasible for the thin form factors of most modern-day mobile phone cameras.
An accurate noise estimate is important when a measure of local similarity is desired. For instance, in denoising, pixels that are similar in value to the pixel currently being denoised are typically averaged together in some fashion. Performance in such denoising operations is directly dependent on the quality of the “similar pixel” set, which in turn is dependent on the “similarity measure.” Robust similarity measures may, therefore, preferably rely on an effective and accurate noise model in order to be able to adapt to imaging conditions. If the noise model is accurate, the similarity measure can help to differentiate between signal and noise. Similarly, in applications where multiple frames are fused (e.g., high dynamic range imaging) or stitching is involved (e.g., panoramic imaging), there may be a need to differentiate between still and moving objects within the images, as well as compensate for registration errors. Relative motion between frames can result from three main factors: 1) object motion; 2) camera shake; and 3) rolling shutter. An accurate noise model, such as that described herein, may help to differentiate between signal and motion for various imaging conditions, moving objects, and hand shake. If the similarity measure is based on an accurate noise model, it will be able to adapt to changing conditions, object motion, and jitter—resulting in fewer “ghosting” artifacts in the image and better overall image quality in a wide variety of imaging conditions.